PrUE: Distilling Knowledge from Sparse Teacher Networks

Although deep neural networks have enjoyed remarkable success across a wide variety of tasks, their ever-increasing size also imposes significant overhead on deployment. To compress these models, knowledge distillation was proposed to transfer from cumbersome (teacher) network into lightweight (student) network. However, guidance teacher does not always improve the generalization students, especially when gap between student and is large. Previous works argued that it due high certainty teacher, resulting in harder labels were difficult fit. soften labels, we present pruning method termed Prediction Uncertainty Enlargement (PrUE) simplify teacher. Specifically, our aims decrease teacher’s about data, thereby generating soft predictions for students. We empirically investigate effectiveness with experiments CIFAR-10/100, Tiny-ImageNet, ImageNet. Results indicate trained sparse teachers achieve better performance. Besides, allows researchers distill deeper students further. Our code made public at: https://github.com/wangshaopu/prue .